Automated program optimization

ABSTRACT

An example of a system may include a processor, and a memory device comprising instructions, which when executed by the processor, cause the processor to access at least one of patient input, clinician input, or automatic input, use the patient input, clinician input, or automatic input in a search method, the search method designed to evaluate a plurality of candidate neuromodulation parameter sets to identify an optimal neuromodulation parameter set, and program a neuromodulator using the optimal neuromodulation parameter set to stimulate a patient.

CLAIM OF PRIORITY

This application is a continuation of U.S. application Ser. No.15/269,674, filed Sep. 19, 2016, which claims the benefit of priorityunder 35 U.S.C. § 119(e) of U.S. Provisional Patent Application Ser. No.62/221,335, filed on Sep. 21, 2015, each of which is herein incorporatedby reference in its entirety.

TECHNICAL FIELD

This document relates generally to medical devices, and moreparticularly, to systems, devices, and methods for delivering neuralmodulation.

BACKGROUND

Neurostimulation, also referred to as neuromodulation, has been proposedas a therapy for a number of conditions. Examples of neurostimulationinclude Spinal Cord Stimulation (SCS), Deep Brain Stimulation (DBS),Peripheral Nerve Stimulation (PNS), and Functional ElectricalStimulation (FES). Implantable neurostimulation systems have beenapplied to deliver such a therapy. An implantable neurostimulationsystem may include an implantable neurostimulator, also referred to asan implantable pulse generator (IPG), and one or more implantable leadseach including one or more electrodes. The implantable neurostimulatordelivers neurostimulation energy through one or more electrodes placedon or near a target site in the nervous system. An external programmingdevice is used to program the implantable neurostimulator withstimulation parameters controlling the delivery of the neurostimulationenergy.

The neurostimulation energy may be delivered in the form of electricalneurostimulation pulses. The delivery is controlled using stimulationparameters that specify spatial (where to stimulate), temporal (when tostimulate), and informational (patterns of pulses directing the nervoussystem to respond as desired) aspects of a pattern of neurostimulationpulses. Many current neurostimulation systems are programmed to deliverperiodic pulses with one or a few uniform waveforms continuously or inbursts. However, neural signals may include more sophisticated patternsto communicate various types of information, including sensations ofpain, pressure, temperature, etc.

Recent research has shown that the efficacy and efficiency of certainneurostimulation therapies can be improved, and their side-effects canbe reduced, by using patterns of neurostimulation pulses that emulatenatural patterns of neural signals observed in the human body.

SUMMARY

Example 1 includes subject matter (such as a device, apparatus, ormachine) comprising: a processor; and a memory device comprisinginstructions, which when executed by the processor, cause the processorto: access at least one of: patient input, clinician input, or automaticinput; use the patient input, clinician input, or automatic input in asearch method, the search method designed to evaluate a plurality ofcandidate neuromodulation parameter sets to identify an optimalneuromodulation parameter set of the plurality of candidateneuromodulation parameter sets; and program a neuromodulator using theoptimal neuromodulation parameter set to stimulate a patient.

In Example 2, the subject matter of Example 1 may include, wherein thepatient input comprises subjective data.

In Example 3, the subject matter of any one of Examples 1 to 2 mayinclude, wherein the subjective data comprises a visual analog scale(VAS), numerical rating scale (NRS), a satisfaction score, a globalimpression of change, or an activity level.

In Example 4, the subject matter of any one of Examples 1 to 3 mayinclude, wherein the clinician input comprises a selectedneuromodulation parameter set, a selected neuromodulation parameter setdimension, or a search method configuration option.

In Example 5, the subject matter of any one of Examples 1 to 4 mayinclude, wherein the selected neuromodulation parameter set dimensioncomprises a spatial location, a frequency, a pulse width, a number ofpulses within a burst or train of pulses, the train-to-train interval,the burst frequency of these trains, a pulse duty cycle, or a burst dutycycle.

In Example 6, the subject matter of any one of Examples 1 to 5 mayinclude, wherein the search method configuration option comprises a testrange for a neuromodulation parameter set dimension, a terminationcriteria for a neuromodulation parameter set test, an amount of time totest a neuromodulation parameter set, a minimum evaluation time for acandidate neuromodulation parameter set, or a survival threshold for aneuromodulation parameter set under test.

In Example 7, the subject matter of any one of Examples 1 to 6 mayinclude, wherein the automatic input comprises data received from apatient device.

In Example 8, the subject matter of any one of Examples 1 to 7 mayinclude, wherein the patient device comprises an accelerometer and theautomatic input comprises activity data.

In Example 9, the subject matter of any one of Examples 1 to 8 mayinclude, wherein the patient device comprises a heart rate monitor andthe automatic input comprises heart rate or heart rate variability.

In Example 10, the subject matter of any one of Examples 1 to 9 mayinclude, wherein the patient device comprises an implantable pulsegenerator and the automatic input comprises field potentials.

In Example 11, the subject matter of any one of Examples 1 to 10 mayinclude, wherein the search method comprises a sorting algorithm thatuses scoring from the patient to sort the plurality of candidateparameter sets and remove parameter sets from the plurality of candidateparameter sets that fail to meet a threshold score.

In Example 12, the subject matter of any one of Examples 1 to 11 mayinclude, wherein the search method comprises a gradient descent systemthat progresses through the plurality of candidate parameter sets tooptimize a dimension of the candidate parameter sets.

In Example 13, the subject matter of any one of Examples 1 to 12 mayinclude, wherein the search method comprises a sensitivity analysis thatbuilds a model from stimulation variables and outcome variables, anduses a regression model to identify a vector of coefficients.

Example 14 includes a machine-readable medium including instructions,which when executed by a machine, cause the machine to performoperations of any of the claims 1-13.

Example 16 includes subject matter (such as a device, apparatus, ormachine) comprising: a processor; and a memory device comprisinginstructions, which when executed by the processor, cause the processorto: access at least one of: patient input, clinician input, or automaticinput; use the patient input, clinician input, or automatic input in asearch method, the search method designed to evaluate a plurality ofcandidate neuromodulation parameter sets to identify an optimalneuromodulation parameter set of the plurality of candidateneuromodulation parameter sets; and program a neuromodulator using theoptimal neuromodulation parameter set to stimulate a patient.

In Example 17, the subject matter of Example 16 may include, wherein thepatient input comprises subjective data.

In Example 18, the subject matter of any one of Examples 16 to 17 mayinclude, wherein the subjective data comprises a visual analog scale(VAS), numerical rating scale (NRS), a satisfaction score, a globalimpression of change, or an activity level.

In Example 19, the subject matter of any one of Examples 16 to 18 mayinclude, wherein the clinician input comprises a selectedneuromodulation parameter set, a selected neuromodulation parameter setdimension, or a search method configuration option.

In Example 20, the subject matter of any one of Examples 16 to 19 mayinclude, wherein the selected neuromodulation parameter set dimensioncomprises a spatial location, a frequency, a pulse width, a number ofpulses within a burst or train of pulses, the train-to-train interval,the burst frequency of these trains, a pulse duty cycle, or a burst dutycycle.

In Example 21, the subject matter of any one of Examples 16 to 20 mayinclude, wherein the search method configuration option comprises a testrange for a neuromodulation parameter set dimension, a terminationcriteria for a neuromodulation parameter set test, an amount of time totest a neuromodulation parameter set, a minimum evaluation time for acandidate neuromodulation parameter set, or a survival threshold for aneuromodulation parameter set under test.

In Example 22, the subject matter of any one of Examples 16 to 21 mayinclude, wherein the automatic input comprises data received from apatient device.

In Example 23, the subject matter of any one of Examples 16 to 22 mayinclude, wherein the patient device comprises an accelerometer and theautomatic input comprises activity data.

In Example 24, the subject matter of any one of Examples 16 to 23 mayinclude, wherein the patient device comprises a heart rate monitor andthe automatic input comprises heart rate or heart rate variability.

In Example 25, the subject matter of any one of Examples 16 to 24 mayinclude, wherein the patient device comprises an implantable pulsegenerator and the automatic input comprises field potentials.

In Example 26, the subject matter of any one of Examples 16 to 25 mayinclude, wherein the search method comprises a sorting algorithm thatuses scoring from the patient to sort the plurality of candidateparameter sets and remove parameter sets from the plurality of candidateparameter sets that fail to meet a threshold score.

In Example 27, the subject matter of any one of Examples 16 to 26 mayinclude, wherein the search method comprises a gradient descent systemthat progresses through the plurality of candidate parameter sets tooptimize a dimension of the candidate parameter sets.

In Example 28, the subject matter of any one of Examples 16 to 27 mayinclude, wherein the search method comprises a sensitivity analysis thatbuilds a model from stimulation variables and outcome variables, anduses a regression model to identify a vector of coefficients.

Example 29 includes subject matter (such as a method, means forperforming acts, machine readable medium including instructions thatwhen performed by a machine cause the machine to performs acts, or anapparatus to perform) comprising: accessing, at a computerized system,at least one of: patient input, clinician input, or automatic input;using the patient input, clinician input, or automatic input in a searchmethod, the search method designed to evaluate a plurality of candidateneuromodulation parameter sets to identify an optimal neuromodulationparameter set of the plurality of candidate neuromodulation parametersets; and programming a neuromodulator using the optimal neuromodulationparameter set to stimulate a patient.

In Example 30, the subject matter of Example 29 may include, wherein thepatient input comprises subjective data, the subjective data a visualanalog scale (VAS), numerical rating scale (NRS), a satisfaction score,a global impression of change, or an activity level.

In Example 31, the subject matter of any one of Examples 29 to 30 mayinclude, wherein the clinician input comprises a selectedneuromodulation parameter set, a selected neuromodulation parameter setdimension, or a search method configuration option.

In Example 32, the subject matter of any one of Examples 29 to 31 mayinclude, wherein the search method comprises a sorting algorithm thatuses scoring from the patient to sort the plurality of candidateparameter sets and remove parameter sets from the plurality of candidateparameter sets that fail to meet a threshold score.

Example 35 includes subject matter (such as a computer-readable medium)comprising: access at least one of: patient input, clinician input, orautomatic input; use the patient input, clinician input, or automaticinput in a search method, the search method designed to evaluate aplurality of candidate neuromodulation parameter sets to identify anoptimal neuromodulation parameter set of the plurality of candidateneuromodulation parameter sets; and program a neuromodulator using theoptimal neuromodulation parameter set to stimulate a patient.

This Summary is an overview of some of the teachings of the presentapplication and not intended to be an exclusive or exhaustive treatmentof the present subject matter. Further details about the present subjectmatter are found in the detailed description and appended claims. Otheraspects of the disclosure will be apparent to persons skilled in the artupon reading and understanding the following detailed description andviewing the drawings that form a part thereof, each of which are not tobe taken in a limiting sense. The scope of the present disclosure isdefined by the appended claims and their legal equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments are illustrated by way of example in the figures ofthe accompanying drawings. Such embodiments are demonstrative and notintended to be exhaustive or exclusive embodiments of the presentsubject matter.

FIG. 1 illustrates a portion of a spinal cord.

FIG. 2 illustrates, by way of example, an embodiment of aneuromodulation system.

FIG. 3 illustrates, by way of example, an embodiment of a modulationdevice, such as may be implemented in the neuromodulation system of FIG.2 .

FIG. 4 illustrates, by way of example, an embodiment of a programmingdevice, such as may be implemented as the programming device in theneuromodulation system of FIG. 2 .

FIG. 5 illustrates, by way of example, an implantable neuromodulationsystem and portions of an environment in which system may be used.

FIG. 6 illustrates, by way of example, an embodiment of an SCS system.

FIG. 7 illustrates, by way of example, an embodiment of data and controlflow in a system that utilizes machine learning to optimizeneurostimulation patterns.

FIGS. 8A-8B illustrate, by way of example, another embodiment of dataand control flow in a system that utilizes machine learning to optimizeneurostimulation patterns.

FIG. 9 illustrates, by way of example, an embodiment of constructingstimulation waveforms in space and time domains.

FIG. 10 illustrates, by way of example, an embodiment of a system thatutilizes machine learning to optimize neurostimulation patterns.

FIG. 11 illustrates, by way of example, an embodiment of a method thatutilizes machine learning to optimize neurostimulation patterns.

FIG. 12 illustrates, by way of example, an embodiment of a system thatutilizes a search method to search for an optimal neuromodulationparameter set.

FIG. 13 illustrates, by way of example, an embodiment of a method thatutilizes a search method to search for an optimal neuromodulationparameter set.

FIG. 14 is a block diagram illustrating a machine in the example form ofa computer system, within which a set or sequence of instructions may beexecuted to cause the machine to perform any one of the methodologiesdiscussed herein, according to an example embodiment.

FIG. 15 illustrates, by way of example, an embodiment of a method thatidentifies parameter sets.

FIG. 16 illustrates, by way of example, an embodiment of a gradientdescent method.

DETAILED DESCRIPTION

The following detailed description of the present subject matter refersto the accompanying drawings which show, by way of illustration,specific aspects and embodiments in which the present subject matter maybe practiced. These embodiments are described in sufficient detail toenable those skilled in the art to practice the present subject matter.Other embodiments may be utilized and structural, logical, andelectrical changes may be made without departing from the scope of thepresent subject matter. References to “an”, “one”, or “various”embodiments in this disclosure are not necessarily to the sameembodiment, and such references contemplate more than one embodiment.The following detailed description is, therefore, not to be taken in alimiting sense, and the scope is defined only by the appended claims,along with the full scope of legal equivalents to which such claims areentitled.

Various embodiments described herein involve spinal cord modulation. Abrief description of the physiology of the spinal cord and relatedapparatus is provided herein to assist the reader. FIG. 1 illustrates,by way of example, a portion of a spinal cord 100 including white matter101 and gray matter 102 of the spinal cord. The gray matter 102 includescell bodies, synapse, dendrites, and axon terminals. Thus, synapses arelocated in the gray matter. White matter 101 includes myelinated axonsthat connect gray matter areas. A typical transverse section of thespinal cord includes a central “butterfly” shaped central area of graymatter 102 substantially surrounded by an ellipse-shaped outer area ofwhite matter 101. The white matter of the dorsal column (DC) 103includes mostly large myelinated axons that form afferent fibers thatrun in an axial direction. The dorsal portions of the “butterfly” shapedcentral area of gray matter are referred to as dorsal horns (DH) 104. Incontrast to the DC fibers that run in an axial direction, DH fibers canbe oriented in many directions, including perpendicular to thelongitudinal axis of the spinal cord. Examples of spinal nerves 105 arealso illustrated, including a dorsal root (DR) 105, dorsal root ganglion107 and ventral root 108. The dorsal root 105 mostly carries sensorysignals into the spinal cord, and the ventral root functions as anefferent motor root. The dorsal and ventral roots join to form mixedspinal nerves 105.

SCS has been used to alleviate pain. A therapeutic goal for conventionalSCS programming has been to maximize stimulation (i.e., recruitment) ofthe DC fibers that run in the white matter along the longitudinal axisof the spinal cord and minimal stimulation of other fibers that runperpendicular to the longitudinal axis of the spinal cord (dorsal rootfibers, predominantly), as illustrated in FIG. 1 . The white matter ofthe DC includes mostly large myelinated axons that form afferent fibers.While the full mechanisms of pain relief are not well understood, it isbelieved that the perception of pain signals is inhibited via the gatecontrol theory of pain, which suggests that enhanced activity ofinnocuous touch or pressure afferents via electrical stimulation createsinterneuronal activity within the DH of the spinal cord that releasesinhibitory neurotransmitters (Gamma-Aminobutyric Acid (GABA), glycine),which in turn, reduces the hypersensitivity of wide dynamic range (WDR)sensory neurons to noxious afferent input of pain signals traveling fromthe dorsal root (DR) neural fibers that innervate the pain region of thepatient, as well as treating general WDR ectopy. Consequently, the largesensory afferents of the DC nerve fibers have been targeted forstimulation at an amplitude that provides pain relief. Currentimplantable neuromodulation systems typically include electrodesimplanted adjacent, i.e., resting near, or upon the dura, to the dorsalcolumn of the spinal cord of the patient and along a longitudinal axisof the spinal cord of the patient.

Activation of large sensory DC nerve fibers also typically creates theparesthesia sensation that often accompanies standard SCS therapy. Someembodiments deliver therapy where the delivery of energy is perceptibledue to sensations such as paresthesia. Although alternative orartifactual sensations, such as paresthesia, are usually toleratedrelative to the sensation of pain, patients sometimes report thesesensations to be uncomfortable, and therefore, they can be considered anadverse side-effect to neuromodulation therapy in some cases. Someembodiments deliver sub-perception therapy that is therapeuticallyeffective to treat pain, for example, but the patient does not sense thedelivery of the modulation field (e.g. paresthesia). Sub-perceptiontherapy may include higher frequency modulation (e.g. about 1500 Hz orabove) of the spinal cord that effectively blocks the transmission ofpain signals in the afferent fibers in the DC. Some embodiments hereinselectively modulate DH tissue or DR tissue over DC tissue to providesub-perception therapy. For example, the selective modulation may bedelivered at frequencies less than 1,200 Hz. The selective modulationmay be delivered at frequencies less than 1,000 Hz in some embodiments.In some embodiments, the selective modulation may be delivered atfrequencies less than 500 Hz. In some embodiments, the selectivemodulation may be delivered at frequencies less than 350 Hz. In someembodiments, the selective modulation may be delivered at frequenciesless than 130 Hz. The selective modulation may be delivered at lowfrequencies (e.g. as low as 2 Hz). The selective modulation may bedelivered even without pulses (e.g. 0 Hz) to modulate some neuraltissue. By way of example and not limitation, the selective modulationmay be delivered within a frequency range selected from the followingfrequency ranges: 2 Hz to 1,200 Hz; 2 Hz to 1,000 Hz, 2 Hz to 500 Hz; 2Hz to 350 Hz; or 2 Hz to 130 Hz. Systems may be developed to raise thelower end of any these ranges from 2 Hz to other frequencies such as, byway of example and not limitation, 10 Hz, 20 Hz, 50 Hz or 100 Hz. By wayof example and not limitation, it is further noted that the selectivemodulation may be delivered with a duty cycle, in which stimulation(e.g. a train of pulses) is delivered during a Stimulation ON portion ofthe duty cycle, and is not delivered during a Stimulation OFF portion ofthe duty cycle. By way of example and not limitation, the duty cycle maybe about 10%±5%, 20%±5%, 30%±5%, 40%±5%, 50%±5% or 60%±5%. For example,a burst of pulses for 10 ms during a Stimulation ON portion followed by15 ms without pulses corresponds to a 40% duty cycle.

FIG. 2 illustrates an embodiment of a neuromodulation system. Theillustrated system 210 includes electrodes 211, a modulation device 212,and a programming device 213. The electrodes 211 are configured to beplaced on or near one or more neural targets in a patient. Themodulation device 212 is configured to be electrically connected toelectrodes 211 and deliver neuromodulation energy, such as in the formof electrical pulses, to the one or more neural targets thoughelectrodes 211. The delivery of the neuromodulation is controlled byusing a plurality of modulation parameters, such as modulationparameters specifying the electrical pulses and a selection ofelectrodes through which each of the electrical pulses is delivered. Invarious embodiments, at least some parameters of the plurality ofmodulation parameters are programmable by a user, such as a physician orother caregiver. The programming device 213 provides the user withaccessibility to the user-programmable parameters. In variousembodiments, the programming device 213 is configured to becommunicatively coupled to modulation device via a wired or wirelesslink. In various embodiments, the programming device 213 includes agraphical user interface (GUI) 214 that allows the user to set and/oradjust values of the user-programmable modulation parameters.

FIG. 3 illustrates an embodiment of a modulation device 312, such as maybe implemented in the neuromodulation system 210 of FIG. 2 . Theillustrated embodiment of the modulation device 312 includes amodulation output circuit 315 and a modulation control circuit 316.Those of ordinary skill in the art will understand that theneuromodulation system 210 may include additional components such assensing circuitry for patient monitoring and/or feedback control of thetherapy, telemetry circuitry and power. The modulation output circuit315 produces and delivers neuromodulation pulses. The modulation controlcircuit 316 controls the delivery of the neuromodulation pulses usingthe plurality of modulation parameters. The lead system 317 includes oneor more leads each configured to be electrically connected to modulationdevice 312 and a plurality of electrodes 311-1 to 311-N distributed inan electrode arrangement using the one or more leads. Each lead may havean electrode array consisting of two or more electrodes, which also maybe referred to as contacts. Multiple leads may provide multipleelectrode arrays to provide the electrode arrangement. Each electrode isa single electrically conductive contact providing for an electricalinterface between modulation output circuit 315 and tissue of thepatient, where N≥2. The neuromodulation pulses are each delivered fromthe modulation output circuit 315 through a set of electrodes selectedfrom the electrodes 311-1 to 311-N. The number of leads and the numberof electrodes on each lead may depend on, for example, the distributionof target(s) of the neuromodulation and the need for controlling thedistribution of electric field at each target. In one embodiment, by wayof example and not limitation, the lead system includes two leads eachhaving eight electrodes.

The neuromodulation system may be configured to modulate spinal targettissue or other neural tissue. The configuration of electrodes used todeliver electrical pulses to the targeted tissue constitutes anelectrode configuration, with the electrodes capable of beingselectively programmed to act as anodes (positive), cathodes (negative),or left off (zero). In other words, an electrode configurationrepresents the polarity being positive, negative, or zero. Otherparameters that may be controlled or varied include the amplitude, pulsewidth, and rate (or frequency) of the electrical pulses. Each electrodeconfiguration, along with the electrical pulse parameters, can bereferred to as a “modulation parameter set.” Each set of modulationparameters, including fractionalized current distribution to theelectrodes (as percentage cathodic current, percentage anodic current,or off), may be stored and combined into a modulation program that canthen be used to modulate multiple regions within the patient.

The number of electrodes available combined with the ability to generatea variety of complex electrical pulses, presents a huge selection ofavailable modulation parameter sets to the clinician or patient. Forexample, if the neuromodulation system to be programmed has sixteenelectrodes, millions of modulation parameter sets may be available forprogramming into the neuromodulation system. Furthermore, for exampleSCS systems may have thirty-two electrodes which exponentially increasesthe number of modulation parameters sets available for programming. Tofacilitate such selection, the clinician generally programs themodulation parameters sets through a computerized programming system toallow the optimum modulation parameters to be determined based onpatient feedback or other means and to subsequently program the desiredmodulation parameter sets. A closed-loop mechanism may be used toidentify and test modulation parameter sets, receive patient orclinician feedback, and further revise the modulation parameter sets toattempt to optimize stimulation paradigms for pain relief. The patientor clinician feedback may be objective and/or subjective metricsreflecting pain, paresthesia coverage, or other aspects of patientsatisfaction with the stimulation.

FIG. 4 illustrates an embodiment of a programming device 413, such asmay be implemented as the programming device 213 in the neuromodulationsystem of FIG. 2 . The programming device 413 includes a storage device418, a programming control circuit 419, and a GUI 414. The programmingcontrol circuit 419 generates the plurality of modulation parametersthat controls the delivery of the neuromodulation pulses according tothe pattern of the neuromodulation pulses. In various embodiments, theGUI 414 includes any type of presentation device, such as interactive ornon-interactive screens, and any type of user input devices that allowthe user to program the modulation parameters, such as touchscreen,keyboard, keypad, touchpad, trackball, joystick, and mouse. The storagedevice 418 may store, among other things, modulation parameters to beprogrammed into the modulation device. The programming device 413 maytransmit the plurality of modulation parameters to the modulationdevice. In some embodiments, the programming device 413 may transmitpower to the modulation device (e.g., modulation device 312 of FIG. 3 ).The programming control circuit 419 may generate the plurality ofmodulation parameters. In various embodiments, the programming controlcircuit 419 may check values of the plurality of modulation parametersagainst safety rules to limit these values within constraints of thesafety rules.

In various embodiments, circuits of neuromodulation, including itsvarious embodiments discussed in this document, may be implemented usinga combination of hardware, software, and firmware. For example, thecircuit of GUI 414, modulation control circuit 316, and programmingcontrol circuit 419, including their various embodiments discussed inthis document, may be implemented using an application-specific circuitconstructed to perform one or more particular functions or ageneral-purpose circuit programmed to perform such function(s). Such ageneral-purpose circuit includes, but is not limited to, amicroprocessor or a portion thereof, a microcontroller or a portionthereof, and a programmable logic circuit or a portion thereof.

FIG. 5 illustrates, by way of example, an implantable neuromodulationsystem and portions of an environment in which system may be used. Thesystem is illustrated for implantation near the spinal cord. However,neuromodulation system may be configured to modulate other neuraltargets. The system 520 includes an implantable system 521, an externalsystem 522, and a telemetry link 523 providing for wirelesscommunication between implantable system 521 and external system 522.The implantable system 521 is illustrated as being implanted in thepatient's body. The implantable system 521 includes an implantablemodulation device (also referred to as an implantable pulse generator,or IPG) 512, a lead system 517, and electrodes 511. The lead system 517includes one or more leads each configured to be electrically connectedto the modulation device 512 and a plurality of electrodes 511distributed in the one or more leads. In various embodiments, theexternal system 402 includes one or more external (non-implantable)devices each allowing a user (e.g. a clinician or other caregiver and/orthe patient) to communicate with the implantable system 521. In someembodiments, the external system 522 includes a programming deviceintended for a clinician or other caregiver to initialize and adjustsettings for the implantable system 521 and a remote control deviceintended for use by the patient. For example, the remote control devicemay allow the patient to turn a therapy on and off and/or adjust certainpatient-programmable parameters of the plurality of modulationparameters. The remote control device may also provide a mechanism forthe patient to provide feedback on the operation of the implantableneuromodulation system. Feedback may be metrics reflecting perceivedpain, effectiveness of therapies, or other aspects of patient comfort orcondition.

The neuromodulation lead(s) of the lead system 517 may be placedadjacent, e.g., resting near, or upon the dura, adjacent to the spinalcord area to be stimulated. For example, the neuromodulation lead(s) maybe implanted along a longitudinal axis of the spinal cord of thepatient. Due to the lack of space near the location where theneuromodulation lead(s) exit the spinal column, the implantablemodulation device 512 may be implanted in a surgically-made pocketeither in the abdomen or above the buttocks, or may be implanted inother locations of the patient's body. The lead extension(s) may be usedto facilitate the implantation of the implantable modulation device 512away from the exit point of the neuromodulation lead(s).

FIG. 6 illustrates, by way of example, an embodiment of an SCS system600. The SCS system 600 generally comprises a plurality ofneurostimulation leads 12 (in this case, two percutaneous leads 12 a and12 b), an implantable pulse generator (IPG) 14, an external remotecontrol (RC) 16, a User's Programmer (CP) 18, an External TrialStimulator (ETS) 20, and an external charger 22.

The IPG 14 is physically connected via two lead extensions 24 to theneurostimulation leads 12, which carry a plurality of electrodes 26arranged in an array. In the illustrated embodiment, theneurostimulation leads 12 are percutaneous leads, and to this end, theelectrodes 26 are arranged in-line along the neurostimulation leads 12.The number of neurostimulation leads 12 illustrated is two, although anysuitable number of neurostimulation leads 12 can be provided, includingonly one. Alternatively, a surgical paddle lead can be used in place ofone or more of the percutaneous leads. As will also be described infurther detail below, the IPG 14 includes pulse generation circuitrythat delivers electrical stimulation energy in the form of a pulsedelectrical waveform (i.e., a temporal series of electrical pulses) tothe electrode array 26 in accordance with a set of stimulationparameters. The IPG 14 and neurostimulation leads 12 can be provided asan implantable neurostimulation kit, along with, e.g., a hollow needle,a stylet, a tunneling tool, and a tunneling straw.

The ETS 20 may also be physically connected via percutaneous leadextensions 28 or external cable 30 to the neurostimulation lead 12. TheETS 20, which has similar pulse generation circuitry as the IPG 14, alsodelivers electrical stimulation energy in the form of a pulsedelectrical waveform to the electrode array 26 in accordance with a setof stimulation parameters. The major difference between the ETS 20 andthe IPG 14 is that the ETS 20 is a non-implantable device that is usedon a trial basis after the neurostimulation lead 12 has been implantedand prior to implantation of the IPG 14, to test the responsiveness ofthe stimulation that is to be provided. Thus, any functions describedherein with respect to the IPG 14 can likewise be performed with respectto the ETS 20.

The RC 16 may be used to telemetrically control the ETS 20 via abi-directional RF communications link 32. Once the IPG 14 andstimulation leads 12 are implanted, the RC 16 may be used totelemetrically control the IPG 14 via a bi-directional RF communicationslink 34. Such control allows the IPG 14 to be turned on or off and to beprogrammed with different stimulation programs after implantation. Oncethe IPG 14 has been programmed, and its power source has been charged orotherwise replenished, the IPG 14 may function as programmed without theRC 16 being present.

The CP 18 provides the user detailed stimulation parameters forprogramming the IPG 14 and ETS 20 in the operating room and in follow-upsessions. The CP 18 may perform this function by indirectlycommunicating with the IPG 14 or ETS 20, through the RC 16, via an IRcommunications link 36. Alternatively, the CP 18 may directlycommunicate with the IPG 14 or ETS 20 via an RF communications link (notshown).

The external charger 22 is a portable device used to transcutaneouslycharge the IPG 14 via an inductive link 38. Once the IPG 14 has beenprogrammed, and its power source has been charged by the externalcharger 22 or otherwise replenished, the IPG 14 may function asprogrammed without the RC 16 or CP 18 being present.

For the purposes of this specification, the terms “neurostimulator,”“stimulator,” “neurostimulation,” and “stimulation” generally refer tothe delivery of electrical energy that affects the neuronal activity ofneural tissue, which may be excitatory or inhibitory; for example byinitiating an action potential, inhibiting or blocking the propagationof action potentials, affecting changes inneurotransmitter/neuromodulator release or uptake, and inducing changesin neuro-plasticity or neurogenesis of tissue. For purposes of brevity,the details of the RC 16, ETS 20, and external charger 22 will not bedescribed herein.

Application of Automated Programming for Sub-Perception StimulationParameters

Sub-perception neuromodulation therapy is a promising area that maypotentially improve patient outcomes. There are numerous potentialsub-perception therapies, so one problem is determining which of themany possible sub-perception therapies may provide the best outcome forthe patient. The problem is compounded by the fact that it frequentlytakes hours or days for a sub-perception therapy to become effective.Without immediate feedback for the health care provider or manufacturerrepresentative it may involve repeat follow-up visits to identify andfine-tune the optimal sub-perception therapy for each individualpatient. An adaptive learning algorithm that can identify the optimalsub-perception therapy for a patient without requiring intensive healthcare provider or manufacturer interaction would have significantclinical and commercial potential.

Software or hardware in the implantable pulse generator (IPG),programming device, or patient remote control may be used during a trialperiod or after permanent implant to identify the optimal sub-perceptionmodality.

A machine learning system may be used in a closed-loop process todevelop optimized stimulation patterns for pain management. Thestimulation patterns may be modulated in both the time and spacedomains. The stimulation patterns may also be modulated in theinformational domain, which refers to the patterns of pulses. Thelearning system may automatically cycle through different sub-perceptionmodalities (e.g., high frequency, burst, low-mid frequency/lowamplitude, etc.), collect patient feedback through a remote control orother system on each of the modalities, and correlate patient feedbackwith therapy efficacy in order to determine an optimal or improvedtherapy. This allows for improved or optimal sub-perception therapy tobe determined without interaction with the health care provider ormanufacturer representative.

An initial set of stimulation patterns may be generated from a domain ofall available stimulation patterns. The initial set may be obtainedusing one or more machine learning or optimization algorithms to searchfor and identify effective patterns. Alternatively, the initial set maybe provided by a clinician.

In addition to the initial set of parameter settings, one or more rangesfor one or more parameters may be determined by the system or providedby a user (e.g., clinician). The ranges may be for various aspectsincluding amplitude, pulse width, frequency, pulse pattern (e.g.,predetermined or parameterized), cycle on/off properties, spatiallocation of field, spatial extent of field, and the like.

Estimated times for wash-in and wash-out may be provided by a user ordetermined by the system. In an embodiment, the system estimates thewash-in/out to be used from the patient feedback. Accurate estimates forwash-in/out are important because over-estimates (e.g., too long of aperiod) may result in patients having unnecessary pain because theprogramming cycles too slowly, and under-estimates (e.g., too short of aperiod) may result in missing important information because the wash-inhas not occurred. So, in an embodiment, a user is able to set theexpected wash-in/out times for the learning machine algorithm to use. Ina further embodiment, the learning machine algorithm may then begin withthe user estimate and then adjust based on data collected from the useror other sources to revise the estimated wash-in or wash-out times.

In the clinical system, a patient may be provided one or morestimulation patterns, which may be tested by the patient with or withoutclinician supervision. Objective pain metrics, subjective pain metrics,or both objective and subjective pain metrics may be received from thepatient, which are used in the machine learning or optimizationalgorithms to develop further sets of patterns. Objective pain metricsinclude those that are physiologically expressed, such as EEG activity,heart rate, heart rate variability, galvanic skin response, or the like.Subjective pain metrics may be provided by the patient and be expressedas “strong pain,” “lower pain,” or numerically in a range, for example.The pain metrics may be communicated using various communicationmechanisms, such as wireless networks, tethered communication,short-range telemetry, or combinations of such mechanisms. The patientmay manually input some information (e.g., subjective pain scores).

A non-exhaustive list of pain metrics is provided herein. One example ofa pain metric is EEG activity (e.g., Theta activity in the somatosensorycortex and alpha and gamma activity in the prefrontal cortex have beenshown to correlate with pain). Another example pain metric is fMRI(activity in the anterior cingulate cortex and insula have been shown tocorrelate with changes in chronic pain). Another example pain metric isfMRI (activity in the pain matrix, which consists of the thalamus,primary somatosensory cortex, anterior cingulate cortex, prefrontalcortex, and cerebellum and is activated in pain conditions). Anotherexample pain metric is heart rate variability, galvanic skin response,cortisol level, and other measures of autonomic system functioning(autonomic system health has been shown to correlate with pain). Anotherexample pain metric is physical activity (amount of physical activityhas been shown to correlate with pain). Another example pain metric ispain scores (may be inputted through an interface where the patientselects a point on a visual analog scale, or clicks a number on anumerical rating scale). Another example pain metric is quantitativesensory testing [e.g., spatial discrimination (two-point, location,diameter), temporal discrimination, detection threshold (mechanical,thermal, electrical), pain threshold (mechanical, thermal, electrical),temporal summation, thermal grill] (QST measures have been shown tocorrelate with pain). Another example pain metric is somatosensoryevoked potentials, contact heat evoked potentials (these have been shownto be correlated with pain). Another example pain metric is H-reflex,nociceptive flexion reflex (these have been shown to be reduced by SCS).Another example pain metric is conditioned place preference (e.g., inone chamber, stimulate with one paradigm 1, in other chamber, stimulatewith paradigm 2. The chamber where the animal spends the most time winsand continues to the next round). Another example pain metric is localfield potential recordings in the pain matrix (recordings of neuralactivity in these areas are possible with invasive electrodes in apreclinical model).

Some pain metrics are primarily preclinical in nature (e.g., conditionedplace preference and local field potential recordings), while others areprimarily clinical in nature (e.g., pain scores and quantitative sensorytesting). However, it is understood that the pain metrics may beobtained in either preclinical or clinical settings.

Pain metrics may be continuously or repeatedly collected from patientsand fed into the machine learning or optimization algorithms to refineor alter the stimulation patterns. For example, the patients mayinteract with a programmer, remote control, bedside monitor, or otherpatient device to record physical condition, pain, medication dosages,etc. The patient device may be wired or wirelessly connected to thesystem with the machine learning system. This closed-loop mechanismprovides an advantage of reducing the search domain during repeatediterations of the machine learning or optimization algorithm. Byreducing the search domain, a clinician is able to more quickly identifyefficacious patterns and a patient may be subjected to shorterprogramming sessions, which produce less discomfort.

In addition to pain metrics, additional patient feedback may beobtained, such as a satisfaction evaluation, self-reported activity,visual analog scale (VAS), or numerical rating scale (NRS). Otherfeedback and input may be obtained, such as from a user (e.g.,clinician) or sensor values.

The physical system may take on many different forms. Data collectedfrom the patient may be measured using wearable sensors (e.g., heartrate monitor, accelerometer, EEG headset, pulse oximeter, GPS/locationtracker, etc.). The pain metrics and other feedback involving manualinput may be entered via remote control or other external device used bythe patient (e.g. cellular phone).

The algorithm may reside on the CP, the IPG, the ETS, the RC or otherexternal device used by the patient, or in the cloud or remote serversconnected to patient external via Wi-Fi, Bluetooth, cellular data, orother wired/wireless scheme. There may be a GUI on the CP, remotecontrol, or other external device, that enables selection of algorithmas well as manual input. Training of the algorithm may take place in theclinic or in daily life, and may be set to be execute continually oronly at certain times. Optimization data may be stored in the cloud sothat optimized patterns and history can be transferred when the patientmoves from trial to permanent implant and also if the IPG is replaced.

FIG. 7 illustrates, by way of example, an embodiment of data and controlflow in a system that utilizes machine learning to optimizeneurostimulation patterns. One or more inputs 700 may be fed into asearch method 702, which may then provide one or more outputs 704. Theinputs 700 may be generally grouped as patient inputs 700-1, user inputs700-2, and automatic inputs 700-3. Outputs 704 may include programmingparameters or scheduling, reports, and other actions. In addition, auser feedback subsystem 706 may be used to query a user (e.g., apatient) and obtain feedback regarding current, previous, or futureprogramming, pain or quality of life scores, or other information.

Inputs 700 may include patient inputs 700-1. Patient inputs 700-1 mayinclude a wide variety of patient data, such as subjective patient painscores or quality of life scores in the form of a visual analog scale(VAS) or numerical rating scale (NRS), satisfaction scores, a GlobalImpression of Change metric, self-reported activity, or somemathematical combination of such metrics (e.g., a weighted sum of VASand activity).

User inputs 700-2 may include various clinician or specialist inputs,such as programs to evaluate or dimensions to evaluate (e.g., x, y,frequency, pulse width, duty cycle, number of pulses per burst, interand intra-pulse frequency for burst waveforms, etc.). Other user input700-2 may include ranges of a parameter to test or threshold values(e.g., minimum or maximum values for a particular parameter (e.g., up to6.0 mA or from 1.0-5.5 mA). User input 700-2 may also include the amountof time to evaluate a program before moving to another program, apatient option to skip a program (e.g., when the patient feels like theprogram is not being effective and there is still two days left in theevaluation time period), a minimum evaluation time before skipping(either programmatically or patient-initiated skip), a survivalthreshold, or search objectives. The survival threshold may be used todetermine whether based on patient feedback, a particular parameter setis retained or discarded from future consideration. For example, aclinician may set a survival threshold of 7.0, such that if a patientexperiences a VAS score greater than 7.0, then the programmingassociated with the score is discarded from future evaluation.Additional, the search method 702 may be modified to discard theprogramming (e.g., in a genetic algorithm, the programming may beremoved from the chromosome pool so that it is not used in futuregenerations).

A search objective may be provided by the patient or clinician (or otheruser). The search objective may tune the search method 702 to worktoward a program that achieves a desired outcome. One or more objectivesmay be selected. When more than one objective is selective, thecombination of objectives may be weighted or ranked. Examples ofobjectives include, but are not limited to a lower charging frequency,more efficient energy usage, maximize pain relief, usage of paresthesia,absence of paresthesia, specified pain relief threshold (e.g., work toachieve a specified VAS score or comfort level), pain relief in one ormore specified areas (e.g., lower back, foot, leg, etc.), bias towardsspecific wave forms or field shapes (e.g., burst waveforms, high rate,long pulse width, etc.), stimulation at specified vertebral levels, orminimum/maximum search time for search to explore various programsbefore the optimal program is determined (e.g., measured in number ofgenerations in a genetic algorithm).

Other user input 700-2 may be received from the patient via the userfeedback subsystem 706. The user feedback subsystem 706 may interfacewith one or more devices (e.g., the CP, the IPG, the ETS, the RC, oranother external device used by the patient or clinician) and obtainpatient feedback (subjective or objective), sensor data, clinicianfeedback, or the like. Patient feedback may be a “best so far” rating ofrecent programming settings, “worse/same/better” rating than a previousprogram, patient indications of comfort, activity, or mood. Variousobjective data may be obtained as well via the user feedback subsystem706, such as activity, impedance/field potential signature, heart rate,heart rate variability, field potentials, and the like. The objectivedata may be used to determine an objective measurement of pain, asurrogate measurement of comfort or quality of life (e.g., more or lessrecorded activity may be associated with less or more discomfort,respectively), or performance data of a particular programming. The userfeedback system 706 may also collect and communicate patient userinterface actions, such when a patient selectively terminates aprogramming session, skips a programming session, or a rank of two ormore programming sessions provided by the patient. The sorted programsmay be used as input to the search method 702 to select, filter, focus,or otherwise modify the search method 702 in future iterations of thepatient feedback loop.

Automatic inputs 700-3 may include some or all of the objective datathat may be collected via the user feedback subsystem 706 (e.g.,activity, heart rate, etc.). For example, an accelerometer may be usedto identify, classify, and obtain patient posture, activity, or thelike. Other automatic inputs 700-3 may include performance data, such asthe duration of a programming session, the number of time a parameterset was tested and the related patient scores/feedback, etc.

Some or all of the outputs 704 may be fed to the user feedback subsystem706 for presentation to one or more users. For example, a weekly reportof the programming used may be provided to a user (e.g., a patient). Theoutput 704 may be used to drive further processing on the user feedbacksubsystem 706 in order to obtain patient inputs 700-1 or user inputs700-2, which may be fed back into the search method 702 and used todetermine additional outputs 704.

The user feedback subsystem 706 may be hosted at an external device(e.g., a central server or a cloud server) and communicatively coupledto one or more devices, including but not limited to the CP, the IPG,the ETS, or the RC. In addition, or alternatively, the user feedbacksubsystem 706 may be partially or fully hosted at one of the devices,such as the CP or the RC.

The user feedback subsystem 706 may provide a user interface to a user.The user interface (UI) may include various user interface controls,such as a body diagram to allow the user to indicate where pain is felton a body, drop down menus, dialog boxes, check boxes, or other UIcontrols to allow the user to input, modify, record, or otherwise managedata representing a patient's comfort level, activity level, mood, orother general preferences to control the search method 702 or astimulator's operation. The UI may prompt the user for variousinformation, such as a daily survey of whether the current programmingis worse, same, or better than a previous programming. The UI may alsoprovide a user with an interface to sort a number of programs accordingto the patient's preference. An example UI is illustrated in FIGS. 8Aand 8B.

FIGS. 8A and 8B illustrate, by way of example, an embodiment of a userinterface to sort programming. In FIG. 8A, the initial state, a numberof user interface elements 800-1, 800-2, 800-3, 800-4, 800-5 arerepresented in an ordered fashion. The number of user interface elementsin FIG. 8A is five, but it is understood that other numbers of userinterface elements may be used, such as, but not limited to ten, twenty,or fifty. In the first instance 802, the UI elements 800 include P1 andP2, which correspond to the first and second programming sets that thepatient experienced. The patient is provided user interface element 804,which represents the third programming set P3, to insert among theordered UI elements 800. The patient may move the UI element 804 toplace it in order of perceived performance (e.g., preference). In thisexample, the patient opts to place UI element 804 in between UI elements800-1 and 800-2, indicating that the programming P3 was better than P2,but worse than P1. In FIG. 8B, the subsequent state 806, after draggingand dropping the UI element 804 to the selected position, the other UIelements 800 are rearranged around the newly inserted UI element 804. Ifa programming set was in the last position (e.g., 800-5), then it isbumped from the list and the programming set in the fourth position(e.g., 800-4) is demoted to the last position. In addition, the user mayadjust the order of the programming sets by dragging and dropping one UIelement from one position to another, in which case, the other UIelements are rearranged in the new order.

Returning to FIG. 7 , additional user interface controls may be providedto a user (e.g., a clinician or a patient) to set test periods for aprogram, manage wash-in/out time, provide user reports with raw data orcompiled data (e.g., charts, graphs, summaries), or control alternatingprograms. For example, a “best so far” (BSF) program may be used andreused over time to ensure that the patient experiences a goodpercentage of effective stimulation. The BSF program may be used asevery other program, every third program, or in other ways to allow thesystem to test additional programs in between the BSF program.

In an embodiment, the user interface for the patient may obscure some orall of the programming details, in effect blinding the patient from theprogram being used. This may be an option and may be settable by theclinician (e.g., via the CP such that the RC does not display certaininformation). By doing so, the patient may not be biased based oncertain settings or values.

A number of different mechanisms may be used in the search method 702.Examples include, but are not limited to a sorting algorithm, a gradientdescent method, a simplex process, a genetic algorithm, a binary search,and sensitivity analysis. Any of the methods may include a pruningprocess, where programs that do not perform within some satisfactoryrange are removed from future use or consideration in a search. Forexample, a program that uses high frequency and causes major discomfortin a patient may be removed from use in future mutations or crossoversin a genetic algorithm search.

FIG. 9 illustrates, by way of example, an embodiment of a sortingalgorithm used as a search method. In general, the sorting algorithmprovides a series of programs, receives user feedback on each program'sefficacy, and sorts the programs according to the user's feedback. Inthe example embodiment illustrated in FIG. 9 , a patient is scheduledfor four programs 900. The patient may be put on a program, e.g.,Program A, for a period of time. The period may vary or may be fixed.The period may be calculated to ensure that the patient is on theprogram for a sufficient amount of time for wash-in/out before thepatient is to evaluate the program. The patient may provide a score. Inthe example illustrated in FIG. 9 , the score is based on a combinationof activity and NRS. Activity may be rated on a numerical scale, such asthe number of hours a patient was active in a given day or an averagenumber of hours that the patient was active in a given week. Othermetrics of activity are considered to be within the scope of thisdisclosure.

Using the activity metric and the NRS, a score is derived and providedto the sorting algorithm 902 (e.g., search method). The sortingalgorithm 902 may then transition to the next program from the scheduledprograms 900. The parameters for the next program (e.g., Program B) arecommunicated to the patient device (e.g., IPG) and used for a period oftime.

After the evaluation period, the patient scores the program and theresult is communicated back to the sorting algorithm 902 (e.g., from theuser feedback subsystem 706). This evaluation loop may continue throughthe scheduled programs 900 and on to other programs. The sortingalgorithm 902 may maintain a sorted data structure of scores 904, whereprograms are sorted based on the score provided by the patient. Apredefined threshold may be used to discard any programs that fail tomeet a minimum score. In the example illustrated in FIG. 9 , thethreshold is five. As such, Program C and Program B are discarded andPrograms D and A are kept for additional evaluation. For example,Programs D and A may be used together or individually in a subsequentschedule of four programs so that the patient can evaluate them againstother programs. In this example, a higher score reflects a betterprogram. This is fine for composite scores, but for NRS, which usuallyuses lower scores as better (meaning less pain), the comparisons andthresholds may be adapted accordingly.

Another search method 702 is illustrated in FIG. 10 , which illustrates,by way of example, an embodiment of a gradient descent method used as asearch method. In general, the gradient descent method is given anaspect of programming to optimize and an initial value for the aspect.The gradient descent method then searches for the local optimal valuefor the aspect. If the method is used to find a local maximum of thefunction, the procedure is then known as a gradient ascent method. Forthe purposes of this discussion, the term gradient descent will be used,but it is understood that when seeking a local maximum, the termgradient ascent may also be applicable.

FIG. 16 illustrates, by way of example, an embodiment of a gradientdescent method 1600. At block 1602, an initial value of the parameter(s)P is taken. The outcome of the parameter(s) P is evaluated (block 1604).This establishes a starting point for the rest of the evaluations. Atblock 1606, a descent direct (+ or −) is determined based on the resultsfrom adjacent values of P. For example, a value higher than the initialvalue of P is evaluated and a value lower than the initial value of P isevaluated. Based on which value shows a better result, the descentmethod 1600 is set to an increasing direction or a decreasing direction.At block 1608, a step size ΔP is selected. The ΔP may be a value basedon the initial value of P. For example, if the initial value of P is200, the ΔP step size may be 50% of the current value of P, such thatthe ΔP=100. At block 1610, the value of P is updated by the ΔP in thedirection determined at operation block 1606. At block 1612, the outcomeis evaluated using the new value of P. At block 1614, it is determinedwhether the method 1600 has met a stopping criteria (e.g., outcomereaches a local minimum). If the stopping criteria is met, then themethod 1600 ends at block 1616 and the value P is output. Otherwise, themethod 1600 iterates to block 1608 to continue processing.

As an example, in FIG. 10 , the gradient descent method 1000 is provideda program 1002 with an indication to optimize on frequency, beginningwith an initial frequency value of 50 Hz. The gradient descent method1000 begins with the initial values and the patient's device isprogrammed with the initial values (e.g., 50 Hz frequency). The patientmay be progressed through a number of programs that are guided by thegradient descent method 1000.

In the example illustrated in FIG. 10 , the gradient descent method 1000begins with a 50 Hz frequency, relates it to a score obtained from thepatient via patient feedback, and then progresses to the next program,which may use a step value of the initial value. Other step values maybe used, such as a fixed number (e.g., increase/decrease by 10 Hz),percentage (e.g., increase/decrease by 100% of the initial value), orother mechanisms. As illustrated, the step value is used until a scoreis received that indicates a decreased program performance. Inparticular, the patient is first presented with a program using 50 Hzfrequency, which is scored a five. Then the patient is presented with aprogram using a 100 Hz frequency, which is scored at a higher score ofsix. Then, using the same step value of 50 Hz, the patient is presentedwith a program using a 150 Hz frequency, which is scored at a lowerscore of three. At this point, the gradient descent method 1000 seeks tofind the local optimal value between the two values of 100 Hz and 150Hz. Continuing with the example, the gradient descent method 1000 tests125 Hz and finds that it is better than 150 Hz, so it continues reducingthe Hz value to seek the next point of deflection (where the scoringtrend changes). The gradient descent method 1000 reduces the Hz to 110Hz and receives a score of seven, which is an increase over the score at125 Hz. The gradient descent method 1000 then tests a value between 100Hz and 110 Hz (e.g., 105 Hz) and may continue in this manner.

The gradient descent method 1000 may maintain the optimal frequencybased on the best observed score. In this case, it is a frequency of 110Hz with a score of seven. The data used in gradient descent method 1000may also be represented in various reports, such as a line graph, barchart, or the like to visualize the scoring trends versus the metricsbeing optimized.

Another search method 702 is sensitivity analysis. Sensitivity analysiscan be used to find regions in a space of input factor for which a modeloutput is either maximum or minimum or meets some optimum criteria. Inan embodiment, a number N of sets of data may be collected thatrepresent stimulation parameters and corresponding outcomes. Thestimulation parameters may include location, amplitude, pulse width,frequency, duty cycle, pattern, fractionalization, etc. Outcomevariables may include visual analog scale (VAS), numerical rating scale(NRS), satisfaction, comfort level, global impression of change,activity, or derived outcome measure, etc. Using the stimulationparameters (X) and the outcome variables (Y), a parameter-outcome modelmay be built. In an embodiment, the parameter-outcome model is aregression model where Y=f(X, a), where a is a vector of k unknowncoefficients to be identified. In general, it is better to have N>k.Using the N sets of data, the sensitivity analysis is applied to assessthe significance of the input variables to the output variables. Theresult of the sensitivity analysis is to identify sensitive parametersets and insensitive parameter sets.

It is understood that other methods may be used to determine or classifysensitive/insensitive parameter sets, such as machine learning, neuralnetworks, or guided selection. With guided selection, a user may bepresented with various stimulation parameter sets and may drop theinsensitive parameters, focusing on the sensitive parameter adjustment.Users may have the option to include insensitive parameters during laterstimulation testing. Using sensitivity analysis, a system mayautomatically choose dimensions or recommend default stimulationparameters.

It is understood that in various embodiments, given Y=f(X, a), a may bea matrix or a vector depending on the dimension of X and Y, e.g., X isk×N, and Y is m×N, that is, for each set of X of k stimulationparameters, there is a set of outcome evaluations Y of m outcomevariables. The a would be a coefficient matrix of m×k that representsthe significance of each of the stimulation parameters to each of theoutcome variables.

FIG. 15 illustrates, by way of example, an embodiment of a method 1500that identifies parameter sets. At block 1502, a threshold value forsensitivity/significance evaluation is identified. At block 1504, a setN of k stimulation parameters to be evaluated is collected, wherein X isof k×N. At block 1506, for each of the stimulation parameters, a set ofoutcome evaluation Y of m variables is collected, where Y is of m×N. Atblock 1508, the parameter-outcome model Y=f(X, a) is built, where a isof m×k. At block 1510, the value a is checked against thesensitivity/significance threshold to identify sensitive sets versusinsensitive sets of each of them outcome variables. At block 1512, it isdetermined whether to reduce the parameters for further evaluation. Ifthe outcome of the determination at block 1512 is negative, then atblock 1514, the method 1500 is complete and the parameter sets areoutput (both the sensitive set of parameters and the insensitive set ofparameters). The method 1500 may output both subsets (sensitive set andinsensitive set), so that further processing or actions may be taken onthem, for example, the sensitive set may be used to guide theprogramming of therapeutic stimulation and insensitive set may be usedto guide the creation of variation in stimulation parameters if needed(e.g. to prevent adaptation to fixed stimulation parameters) but doesnot significantly alter the therapeutic effect. If the outcome of thedetermination at block 1512 is affirmative, then for a sensitive set, atblock 1516, the set is kept for further evaluation and the number ofstimulation parameters k is updated. For an insensitive set, at block1518, the insensitive set is dropped from further evaluation, butrecorded for later use (as previously described). The method 1500continues to block 1504 for continued execution. Continued iterations ofthe method 1500 refine the sensitive subset to provide additional levelsof sensitivity analysis.

It is understood that the user may bound the search space in any of thesearch methods discussed in this document. The user may bound one ormore of location bounds, location discreet points, pulse width bounds,frequency bounds, or amplitude bounds.

In addition, it is understood that the search methods may be used todetermine time or space parameters for stimulation. A brief discussionof time and space parameters is provided herein.

For the purposes of this discussion, a stimulation protocol may beconsidered as a construction of building blocks beginning with a pulse.A pulse is single waveform and typically has a timescale in themillisecond range. A burst is a sequence of pulses and may have atimescale on the millisecond to second range. A train is a sequence ofbursts and may have a timescale of millisecond, seconds, or even minutesdepending on the programming used. A programming sequence is acombination of pulses, bursts, and trains. The programming sequence mayalso include pauses; periods with no electrical stimulation. Aprogramming sequence may be cyclical over short durations or benon-cyclical over a short duration, but repeat over some longer“macropulse” duration.

In a pulse burst or a pulse train, the intervals between pulses may beregular or irregular. In general, the time domain includes stimulationparameters that control the timing, size, or shape of pulses. Timedomain parameters include, but are not limited to, the pulse rate, pulseamplitude, pulse shape, pulse width, and interpulse delay (e.g., betweenbursts or trains).

In addition to the characteristics of the pulses, the location anddirection of stimulation may be controlled using stimulation parametersin the space domain. Various spatial domain parameters include, but arenot limited to, lead activation (e.g., which lead(s) areactive/inactive), electrode activation (e.g., which electrode(s) in alead are active/inactive) and active contact fractionalization (e.g., ofthe active electrodes, how much current is supplied to each activeelectrode in a lead).

The search method 702 may search for a best location among a set ofpossible locations to apply stimulation (e.g., spatial domainparameter). As illustrated in FIG. 11 , the search method 702 may beprovided with a starting location S 1100 and then use an optimizationapproach to determine a final location F 1102. Beginning at the startinglocation 1100, the search method 702 may progress through one or moretest points until it finds an optimal point, which is then consideredthe final location 1102. The test points may be defined as a targetvolume of activation (VOA), target field, or target pole, in variousembodiments. Test points may be defined by contact locations. Testpoints may be provided by a user (e.g., user selected test points).Alternatively, the test point target pole may be a virtual or ideal poleor multipole pole (e.g., a virtual bipole or tripole). A virtualmultipole may be defined by computing field potential values that thevirtual multipole creates on an array of spatial observation points, anddetermining the stimulation amplitude distribution on the electrodesthat would result in estimated electrical field potential values at thespatial observation points that best matches the desired field potentialvalues at the spatial observation points. It can be appreciated thatcurrent steering can be implemented by moving the virtual multipolesabout the leads, such that the appropriate stimulation amplitudedistribution for the electrodes is computed for each of the variouspositions of the virtual multipole. As a result, the current steeringcan be implemented using an arbitrary number and arrangement ofelectrodes.

The search method 702 then searches a space for the best point among aset of possible points. The search space may be bounded. In the exampleillustrated in FIG. 11 , the search space is bounded by a left bound1104, a right bound 1106, a rostral bound 1108, and a caudal bound 1110.Bounds may be clinician/user defined or based on a pain drawing (e.g.,using a mapping of probability between pain and ideal stimulationlocus). Alternatively, the search method 702 may define bounds duringthe optimization process (e.g., by using insensitive parameters or painscore thresholding, etc.).

As described above, the search method 702 may program the patient'sdevice (e.g., IPG) with parameters and allow the program to operate fora sufficient time to allow for wash-in/out. Once the patient has someexperience with the program, the patient may provide feedback, which isused as input into the search method 702 and may alter the selection oridentification of other programs. It is understood that any of thesearch or optimization algorithms discussed in this document may be usedto search for optimal spatial parameters. Additionally, the spatialparameters may be searched in combination with time domain parameters.

The search method 702 may use user feedback. It is understood that auser's comfort level may vary day-to-day or hour-to-hour with the sameprogram. The user's posture, activity level, medication, and othervariables may introduce noise into the scores provided by the user. Inorder to manage the noise in these metrics, one or more techniques maybe used. In an embodiment, an overall satisfaction score is used.

As another example, a numerous amount of samples may be taken. Thus, inan embodiment, multiple scores are obtained for a given program. Thescores may be averaged, added, or otherwise mathematically combined toobtain an output score.

As another example, objective measures may be obtained and used forcorroboration with the patient feedback, used in combination with thepatient feedback (e.g., mathematically combined), or used in place ofpatient feedback. Objective measures may include physiological metricssuch as posture, activity, heart rate, heart rate variability, etc.

As another example, repeated evaluations of the same program may be usedto recheck and verify a patient's reaction to a given program. Knowinglyor unknowingly, the patient may be presented with the same program at alater time and the patient's feedback from the program may be averagedor otherwise combined with previous feedback from an earlier instance ofthe program's use.

After identifying a good program (or multiple good programs), the IPGmay be configured to operate with the identified program(s) for a timeuntil the next re-optimization and reconfiguration. Ideally during thistime, the patient has the best program available at that time. However,the patient may experience some neurological conditioning where thepatient's neurological system becomes accustomed to the program in a waythat the program loses some of its effect on the patient. The result maybe the patient experiences more pain or different pain as the days orweeks go by. In an effort to avoid this neurological conditioning, thepatient may be provided with a rotating or shifting program schedule.This altering program schedule may provide for neuroplasticity in thepatient's neurological system; keeping the patient's neurological systemconfused and improving the performance or longevity of a program'sefficacy. Such programming reduces the need to reprogram, therebyreducing power consumption in the devices in the neuromodulation systemand increasing patient quality of life.

Thus, in an embodiment, two or more programs may be cycled on the IPG.The programs may be altered on a regular basis (e.g., changing everyseven days) or on an irregular basis (e.g., changing randomly within afive to seven day range). When more than two programs are cycled, theorder of the programs may be regular (e.g., cycling in sequence) orirregular (e.g., cycling arbitrarily).

In an example, a burst waveform may run for a specified period beforethe system automatically shifts to a high rate wave form, with cyclingbetween programs. To avoid a period where pain relief and/or therapeuticeffect from the first waveforms (A) ends before the therapeutic effectof the next waveform (B) begins, waveform B may be initiated prior tothe termination of waveform A.

FIG. 12 illustrates, by way of example, an embodiment of a system 1200that utilizes a search method to search for an optimal neuromodulationparameter set. The system 1200 may take on one of many forms. The system1200 may be a remote control or other external device used by a patientor clinician. Alternatively, the system 1200 may be a server orcloud-based device, a network appliance, or other networked deviceconnected via a network (or combination of networks) to a user device.The networks may include local, short-range, or long-range networks,such as Bluetooth, cellular, Wi-Fi, or other wired or wireless networks.

The system 1200 includes a processor 1202 and a memory 1204. Theprocessor 1202 may be any single processor or group of processors thatact cooperatively. The memory 1204 may be any type of memory, includingvolatile or non-volatile memory. The memory 1204 may includeinstructions, which when executed by the processor 1202, cause theprocessor 1202 to access at least one of: patient input, clinicianinput, or automatic input.

In an embodiment, the patient input comprises subjective data. Invarious embodiments, the subjective data may include comprises a visualanalog scale (VAS), numerical rating scale (NRS), a satisfaction score,a global impression of change, or an activity level.

In an embodiment, the clinician input includes a selectedneuromodulation parameter set, a selected neuromodulation parameter setdimension, or a search method configuration option. As describedelsewhere in this document, the clinician or other user may select oneor more programs (e.g., parameter sets) to evaluate. Alternatively or inaddition to selecting a program or programs, the clinician or user mayselect a particular dimension of a parameter set. Dimensions includeaspects like frequency or amplitude of a stimulation waveform. Thus, invarious embodiments, the selected neuromodulation parameter setdimension comprises a spatial location, a frequency, a pulse width, anumber of pulses within a burst or train of pulses, the train-to-traininterval, the burst frequency of these trains, a pulse duty cycle, or aburst duty cycle. In addition to programs or dimensions, a user mayconfigure the search method, such as by configuring what the patient cando to modify programs or skip programs, or other aspects of the searchmethodology. Thus, in various embodiments, the search methodconfiguration option comprises a test range for a neuromodulationparameter set dimension, a termination criteria for a neuromodulationparameter set test, an amount of time to test a neuromodulationparameter set, a minimum evaluation time for a candidate neuromodulationparameter set, or a survival threshold for a neuromodulation parameterset under test.

In an embodiment, the automatic input comprises data received from apatient device. Patient devices include various devices, such as apersonal computer (PC), a tablet PC, a hybrid tablet, a personal digitalassistant (PDA), a mobile telephone, an implantable pulse generator(IPG), an external remote control (RC), a User's Programmer (CP), anExternal Trial Stimulator (ETS), or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. In an embodiment, the patient device comprises anaccelerometer and the automatic input comprises activity data. In anembodiment, the patient device comprises a heart rate monitor and theautomatic input comprises heart rate or heart rate variability. In anembodiment, the patient device comprises an implantable pulse generatorand the automatic input comprises field potentials.

The processor 1202 may further use the patient input, clinician input,or automatic input in a search method, the search method designed toevaluate a plurality of candidate neuromodulation parameter sets toidentify an optimal neuromodulation parameter set. As describedelsewhere in this document, various search methods may be used toevaluate a group of programs or progress through programs to identify anoptimal program.

In an embodiment, the search method comprises a sorting algorithm thatuses scoring from the patient to sort the plurality of candidateparameter sets and remove parameter sets from the plurality of candidateparameter sets that fail to meet a threshold score.

In an embodiment, wherein the search method comprises a gradient descentsystem that progresses through the plurality of candidate parameter setsto optimize a dimension of the candidate parameter sets.

In an embodiment, the search method comprises a sensitivity analysisthat builds a model from stimulation variables and outcome variables,and uses a regression model to identify a vector of coefficients.

The processor 1202 may further program a neuromodulator using theoptimal neuromodulation parameter set to stimulate a patient.

FIG. 13 illustrates, by way of example, an embodiment of a method 1300that utilizes a search method to search for an optimal neuromodulationparameter set. At 1302, at least one of: patient input, clinician input,or automatic input is accessed at a computerized system.

In an embodiment, the patient input comprises subjective data. Invarious embodiments, the subjective data comprises a visual analog scale(VAS), numerical rating scale (NRS), a satisfaction score, a globalimpression of change, or an activity level.

In various embodiments, the clinician input comprises a selectedneuromodulation parameter set, a selected neuromodulation parameter setdimension, or a search method configuration option. In a further variousembodiments, the selected neuromodulation parameter set dimensioncomprises a spatial location, a frequency, a pulse width, a number ofpulses within a burst or train of pulses, the train-to-train interval,the burst frequency of these trains, a pulse duty cycle, or a burst dutycycle. In another embodiment, the search method configuration optioncomprises a test range for a neuromodulation parameter set dimension, atermination criteria for a neuromodulation parameter set test, an amountof time to test a neuromodulation parameter set, a minimum evaluationtime for a candidate neuromodulation parameter set, or a survivalthreshold for a neuromodulation parameter set under test.

In an embodiment, the automatic input comprises data received from apatient device. In a further embodiment, the patient device comprises anaccelerometer and the automatic input comprises activity data. Inanother embodiment, the patient device comprises a heart rate monitorand the automatic input comprises heart rate or heart rate variability.In another embodiment, the patient device comprises an implantable pulsegenerator and the automatic input comprises field potentials.

At 1304, the patient input, clinician input, or automatic input is usedin a search method, the search method designed to evaluate a pluralityof candidate neuromodulation parameter sets to identify an optimalneuromodulation parameter set. In an embodiment, the search methodcomprises a sorting algorithm that uses scoring from the patient to sortthe plurality of candidate parameter sets and remove parameter sets fromthe plurality of candidate parameter sets that fail to meet a thresholdscore. In an embodiment, the search method comprises a gradient descentmethod that progresses through the plurality of candidate parameter setsto optimize a dimension of the candidate parameter sets. In anembodiment, the search method comprises a sensitivity analysis thatbuilds a model from stimulation variables and outcome variables, anduses a regression model to identify a vector of coefficients.

At 1306, a neuromodulator is programmed using the optimalneuromodulation parameter set to stimulate a patient.

FIG. 14 is a block diagram illustrating a machine in the example form ofa computer system 1400, within which a set or sequence of instructionsmay be executed to cause the machine to perform any one of themethodologies discussed herein, according to an example embodiment. Inalternative embodiments, the machine operates as a standalone device ormay be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of either a serveror a client machine in server-client network environments, or it may actas a peer machine in peer-to-peer (or distributed) network environments.The machine may a personal computer (PC), a tablet PC, a hybrid tablet,a personal digital assistant (PDA), a mobile telephone, an implantablepulse generator (IPG), an external remote control (RC), a User'sProgrammer (CP), an External Trial Stimulator (ETS), or any machinecapable of executing instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein. Similarly, the term “processor-basedsystem” shall be taken to include any set of one or more machines thatare controlled by or operated by a processor (e.g., a computer) toindividually or jointly execute instructions to perform any one or moreof the methodologies discussed herein.

Example computer system 1400 includes at least one processor 1402 (e.g.,a central processing unit (CPU), a graphics processing unit (GPU) orboth, processor cores, compute nodes, etc.), a main memory 1404 and astatic memory 1406, which communicate with each other via a link 1408(e.g., bus). The computer system 1400 may further include a videodisplay unit 1410, an alphanumeric input device 1412 (e.g., a keyboard),and a user interface (UI) navigation device 1414 (e.g., a mouse). In oneembodiment, the video display unit 1410, input device 1412 and UInavigation device 1414 are incorporated into a touch screen display. Thecomputer system 1400 may additionally include a storage device 1416(e.g., a drive unit), a signal generation device 1418 (e.g., a speaker),a network interface device 1420, and one or more sensors (not shown),such as a global positioning system (GPS) sensor, compass,accelerometer, or other sensor.

The storage device 1416 includes a machine-readable medium 1422 on whichis stored one or more sets of data structures and instructions 1424(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1424 mayalso reside, completely or at least partially, within the main memory1404, static memory 1406, and/or within the processor 1402 duringexecution thereof by the computer system 1400, with the main memory1404, static memory 1406, and the processor 1402 also constitutingmachine-readable media.

While the machine-readable medium 1422 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” mayinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more instructions 1424. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including but not limited to, by way ofexample, semiconductor memory devices (e.g., electrically programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM)) and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

The instructions 1424 may further be transmitted or received over acommunications network 1426 using a transmission medium via the networkinterface device 1420 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), theInternet, mobile telephone networks, plain old telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-Aor WiMAX networks). The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding, orcarrying instructions for execution by the machine, and includes digitalor analog communications signals or other intangible medium tofacilitate communication of such software.

The above detailed description is intended to be illustrative, and notrestrictive. The scope of the disclosure should, therefore, bedetermined with references to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A method for programming a neuromodulator with aselected neuromodulation parameter set, the method comprising:programming the neuromodulator with a schedule of differentneuromodulation parameter sets to deliver sub-perception neuromodulationto a patient; delivering sub-perception neuromodulation to a patientwith the neuromodulator, using each of the different neuromodulationparameter sets in the schedule; identifying the selected neuromodulatorparameter set to be programmed into the neuromodulator by implementing,using a computerized system, an automated, iterative search method withadaptive learning, that is designed to identify the selectedneuromodulation parameter set, the automated, iterative search methodwith adaptive learning including: accessing, using the computerizedsystem, a feedback input indicative of neuromodulation efficacy for eachof the different neuromodulation parameter sets in the schedule, thefeedback input being based on an automatic input derived using a patientsensor output; ranking, using the computerized system and the feedbackinput, the neuromodulation efficacy for each of the differentneuromodulation parameter sets in the schedule to provide evaluationresults for the different neuromodulation parameter sets; determining,using a machine learning algorithm of the computerized system, a revisedschedule of neuromodulation parameter sets based on the evaluationresults from at least one previous schedule, wherein the revisedschedule differs from an immediately preceding schedule by including atleast one different parameter set than was included in the immediatelypreceding schedule; programming, using the computerized system, theneuromodulator with the revised schedule of neuromodulation parametersets; and delivering sub-perception neuromodulation to the patient withthe neuromodulator, using each of the different neuromodulationparameters sets in the revised schedule; iterating the accessing, theranking, the determining, and the programming of the automated,iterative search method with adaptive learning until the evaluationresults indicate that the selected neuromodulation parameter set wasused to deliver sub-perception neuromodulation.
 2. The method of claim1, wherein the different neuromodulation parameter sets include a firstparameter set and a second parameter set, and the deliveringsub-perception neuromodulation includes delivering the sub-perceptionneuromodulation using the first parameter set for at least a wash-intime, and initiating delivery of the sub-perception neuromodulationusing the second parameter set at least after a wash-out time after thesub-perception neuromodulation using the first parameter set hasterminated.
 3. The method of claim 2, wherein the wash-in time orwash-out time is user-provided.
 4. The method of claim 2, wherein thewash-in time or wash-out time is determined using the feedback input. 5.The method of claim 1, wherein the feedback input includes a painmetric.
 6. The method of claim 1, wherein the feedback input furtherincludes a satisfaction evaluation, a self-reported activity, a visualanalog scale, or a numerical rating scale.
 7. The method of claim 1,further comprising receiving a user-provided search objective used toprovide the evaluation results.
 8. The method of claim 1, wherein theranking the neuromodulation efficacy for each of the differentneuromodulation parameter sets in the schedule to provide evaluationresults includes sorting the neuromodulation efficacy for the differentneuromodulation parameter sets.
 9. The method of claim 1, the rankingthe neuromodulation efficacy for each of the different neuromodulationparameter sets includes scoring each of the different neuromodulationparameter sets and discarding one or more parameter sets that fail tomeet a threshold score.
 10. The method of claim 1, wherein the schedulein-includes a rotating or shifting schedule of parameter sets.
 11. Themethod of claim 1, wherein the schedule includes a random cycling of aplurality of parameter sets.
 12. A method for programming aneuromodulator with a selected neuromodulation parameter set, the methodcomprising: programming the neuromodulator with a schedule of differentneuromodulation parameter sets to deliver sub-perception neuromodulationto a patient; delivering sub-perception neuromodulation to a patientwith the neuromodulator using each of the different neuromodulationparameter sets in the schedule; identifying the selected neuromodulatorparameter set to be programmed into the neuromodulator by implementing,using a computerized system, an automated, iterative search method withadaptive learning that includes at least one of a sorting algorithm, agradient descent method, a simplex process, a genetic algorithm, abinary search, or a sensitivity analysis, that is designed to identifythe selected neuromodulation parameter set, the automated, iterativesearch method with adaptive learning including: accessing, using thecomputerized system, a feedback input indicative of neuromodulationefficacy for each of the different neuromodulation parameter sets in theschedule, the feedback input being based on an automatic input derivedusing a patient sensor output; ranking, using the computerized systemand the feedback input, the neuromodulation efficacy for each of thedifferent neuromodulation parameter sets in the schedule to provideevaluation results for the different neuromodulation parameter sets;determining, using a machine learning algorithm of the computerizedsystem, a revised schedule of neuromodulation parameter sets based onthe evaluation results from at least one previous schedule, wherein therevised schedule differs from an immediately preceding schedule byincluding at least one different parameter set than was included in theimmediately preceding schedule; programming, using the computerizedsystem, the neuromodulator with the revised schedule of neuromodulationparameter sets; and delivering sub-perception neuromodulation to thepatient with the neuromodulator, using each of the differentneuromodulation parameters sets in the revised schedule; iterating theaccessing, the ranking, the determining, and the programming of theautomated, iterative search method with adaptive learning until theevaluation results indicate that the selected neuromodulation parameterset was used to deliver sub-perception neuromodulation.
 13. The methodof claim 12, wherein the different neuromodulation parameter setsinclude a first parameter set and a second parameter set, and thedelivering sub-perception neuromodulation includes delivering thesub-perception neuromodulation using the first parameter set for atleast a wash-in time, and initiating delivery of the sub-perceptionneuromodulation using the second parameter set at least after a wash-outtime after the sub-perception neuromodulation using the first parameterset has terminated.
 14. The method of claim 13, wherein the wash-in timeor wash-out time is user-provided.
 15. The method of claim 13, whereinthe wash-in time or wash-out time is determined using the feedbackinput.
 16. The method of claim 12, wherein the schedule in includes arotating or shifting schedule of parameter sets.
 17. The method of claim12, wherein the schedule includes a random cycling of a plurality ofparameter sets.
 18. A method for programming a neuromodulator with aselected neuromodulation parameter set, including a selectedfractionalized current distribution, configured for delivering stimulusenergy at a corresponding selected stimulus locus, the methodcomprising: programming the neuromodulator with a schedule of differentneuromodulation parameter sets, each including a differentfractionalized current distribution configured for delivering stimulusenergy at corresponding stimulus loci to deliver sub-perceptionneuromodulation to a patient; delivering sub-perception neuromodulationto a patient with the neuromodulator using each of the differentneuromodulation parameter sets in the schedule to deliver stimulusenergy to each of the corresponding stimulus loci; identifying theselected neuromodulator parameter set to be programmed into theneuromodulator by implementing, using a computerized system, anautomated, iterative search method with adaptive learning that isdesigned to identify the selected neuromodulation parameter set,including the selected fractionalized current distribution, configuredfor delivering stimulus energy at the corresponding selected stimuluslocus, the automated, iterative search method with adaptive learningincluding: accessing, using the computerized system, a feedback inputindicative of neuromodulation efficacy for each of the differentneuromodulation parameter sets, each including a differentfractionalized current contribution corresponding stimulus loci in theschedule, the feedback input being based on an automatic input derivedusing a sensor output; ranking, using the computerized system and thefeedback input, the neuromodulation efficacy for each of the differentneuromodulation parameter sets and corresponding stimulus loci in theschedule to provide evaluation results for the different neuromodulationparameter sets; determining, using a machine learning algorithm of thecomputerized system, a revised schedule of neuromodulation parametersets configured for delivering stimulus energy at corresponding stimulusloci based on the evaluation results from at least one previousschedule, wherein the revised schedule differs from an immediatelypreceding schedule by including at least one different parameter setthan was included in the immediately preceding schedule; programming,using the computerized system, the neuromodulator with the revisedschedule of neuromodulation parameter sets configured for deliveringstimulus energy at corresponding stimulus loci; and deliveringsub-perception neuromodulation to the patient with the neuromodulator,using each of the different neuromodulation parameters sets in therevised schedule; iterating the accessing, the evaluating, thedetermining, and the programming of the automated, iterative searchmethod with adaptive learning are continue until the evaluation resultsindicate the selected neuromodulation parameter set, including theselected fractionalized current distribution, was used to deliverneuromodulation to the corresponding selected stimulus locus.
 19. Themethod of claim 18, further comprising bounding a space for searchingfor the selected stimulus locus that corresponds to the selectedneuromodulation parameter set, wherein the determining the revisedschedule includes determining neuromodulation parameter sets withfractionalized current contributions that have corresponding stimulusloci within the space.
 20. The method of claim 18, wherein the rankingthe neuromodulation efficacy for each of the different neuromodulationparameter sets includes scoring each of the different neuromodulationparameter sets and discarding one or more parameter sets that fail tomeet a threshold score.